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Abstract 
Evaluation capacity building (ECB) is a critical aspect of continuous program improvement. At 
community and technical colleges (CTCs), this is particularly important given the many ways in 
which CTCs must use data to meet reporting requirements. ECB helps CTC staff to use and 
interpret data that goes beyond reporting to create a system that enables for program 
improvement through constant monitoring, assessment, and modification. This paper aims to 
advance how CTCs can include ECB into their institutions’ practices based on the authors’ 
experiences of evaluating eight Trade Adjustment Assistance Community College and Career 
Training (TAACCCT) programs at 14 CTCs across the United States. Based on these 
experiences, the authors provide an ECB planning tool and a series of recommendations for ECB 
practitioners who work with CTCs to integrate ECB concepts from a project’s planning stages, 
through grant implementation and planning for sustainability after a project’s primary funding 


source ends. 
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Evaluation Practice Within the Context of Community and Technical Colleges 

Engaging in the process of evaluation is an important part of a continuous improvement 
model (Huffman, Thomas, & Lawrenz, 2008). In community and technical colleges (CTCs), 
elements of evaluation are woven into several facets of colleges’ roles and expectations, 
including federal reporting requirements, accreditation, and national-level voluntary initiatives. 
For instance, CTCs participating in federal student financial aid programs are required to report 
data (e.g., enrollments, program completions, graduation rates, student financial aid provided, 
faculty and staff numbers) to the U.S. Department of Education’s National Center for Education 
Statistics Integrated Postsecondary Education Data System (NCES IPEDS). Through the process 
of institutional and program accreditation, CTC faculty members engage in self-assessments that 
necessitate the collection and review of data and the faculty evaluate their program against a set 
of standards. Within the last 15 years, the use of student learning outcomes has become one of 
the core accreditation requirements for program improvement purposes (Stitt-Bergh, 2016). 
Consequently, faculty and staff participating in accreditation have needed to become more 
familiar with approaches to measure, collect, and report on student learning outcomes. Lastly, 
over the past decade CTCs have faced growing pressure to be accountable to the public. The 
impetus for this scrutiny came from a report released in 2006 by then-Education Secretary 
Margaret Spellings’ Commission on the Future of Higher Education (Boggs, 2009). 
Consequently, a coalition of CTC organizations with support from the Bill & Melinda Gates 
Foundation and Lumina Foundation developed the Voluntary Framework of Accountability, 
which includes various measures of institutional effectiveness (Ewell, 2011). A second widely 
known CTC reform initiative initiated by the Lumina Foundation and several other partners is 
Achieving the Dream (ATD). A central strategy of ATD is for participating colleges to collect 


and use data to assess student progress, identify barriers to student success, and develop 


strategies to address the barriers. Essentially the ATD initiative is about building a culture of 
evidence within the college to provide programs and services. 
Statement of the Problem 

While CTC faculty and staff have become familiar with the evaluation requirements for 
federal reporting and accreditation purposes, most are still unfamiliar with the evaluation process 
of grant-funded initiatives (Yarrow, 2016). One possible explanation is that federally funded 
grant programs often require grantees to hire a third-party evaluator to conduct a rigorous 
evaluation; hence, grantees rely on their evaluator to collect, analyze, and report data to their 
funding agencies. CTCs have limited evaluation resources and, as a result, evaluation capacity 
building (ECB) with program staff is often overlooked. The evaluation effort tends to dissipate 
after the grant ends without having built program and institutional staff's evaluation capacity— 
something that could have occurred throughout the grant period. In the absence of data to 
continuously embrace accountability as part of staffs culture, fidelity of implementation is likely 
to fade away after the grant. The effect and sustainability of the program then becomes 
questionable. The authors’ review of the literature revealed very few accounts of documented 
efforts focused on building higher education institutions’ evaluation capacity. The few studies 
that reported ECB examples tended to revolve around efforts to support faculty and staff to 
prepare for accreditation (Stitt-Bergh, 2016; Parsons, Lovato, Hutchinson, & Wilson, 2016). For 
instance, Parsons and colleagues established a community of learners to promote evaluative 
inquiry within a community college setting and applied the model in the college’s accreditation 
review process. The research found that the model utilized was successful, but the model was 


eventually eliminated due to budget cuts. 


This paper has a twofold purpose: First, it aims to advance ECB literature by sharing the 
authors’ and their organization’s collective evaluation experiences with eight Trade Adjustment 
Assistance Community College and Career (TAACCCT) programs at 14 CTCs across 10 states 
of the United States. Second, it provides an ECB planning tool and recommendations for ECB 
practitioners who work with CTCs that are seeking opportunities and resources for ECB through 
the implementation of a grant. In the following sections, the authors first provide an overview of 
the ECB conceptual framework and discuss the theory of action supporting ECB. Then, the 
authors describe the purpose of TAACCCT grants and the evolution of evaluation requirements 
over the course of four rounds of a grant competition to set up the context for their ECB work 
within TAACCCT grant programs. Specifically, the authors discuss their evaluation approaches 
to embed ECB work within their evaluation processes and address the challenges encountered. 
The paper concludes with an ECB planning tool. 

ECB Conceptual Framework 

ECB is intentional work that is driven by the needs and motivation of the individual or 
organization who initiates the work (Preskill & Boyle, 2008; Stockdill, Baizerman, & Compton, 
2002). In general, the short-term goal of ECB is to acquire knowledge, skills, and attitudes to 
conduct quality evaluation, and the long-term goal is to internalize the evaluation processes and 
practices as a part of the organization operations. As stated by Stockdill, Baizerman, and 
Compton (2002), the goal of ECB is to “create and sustain overall organizational processes that 
make quality evaluation and its uses routine” (p. 14). Preskill and Boyle (2008) synthesized the 
ECB literature and developed a multidisciplinary framework that provides guidelines for ECB 
efforts. Figure 1 is the theory of action of a typical ECB work derived from Preskill and Boyle’s 


framework. 


Figure 1. ECB Theory of Action Logic Model 
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According to Preskill and Boyle (2008), to ascertain successful, feasible, and sustainable 
ECB, an ideal ECB design starts with an initial assessment of the organization’s learning 
capacity, including leadership support, evaluation culture, systems and structures within the 
organization that support evaluation activities, and communication channels to access and 
disseminate evaluation information. ECB design also requires a full understanding of the 
motivations, assumptions, and expectations of the ECB efforts to help determine who should 
participate and which ECB strategies might be most appropriate. Once ECB strategies are 
chosen, it is essential for ECB practitioners to ensure the strategies were implemented as planned 
and participants are actively involved in the effort. If these elements all take place as planned 
(i.e., inputs, strategies, and outputs), ECB participants should demonstrate positive change in 
terms of knowledge, skills, and attitudes toward evaluation. When ECB is a part of 
organizational learning capacity, changes made at the individual level can lead to changes at the 
organizational level in the long term, which makes evaluative thinking and practice routine. This 
experience will allow program staff to understand how to collect and use data for continuous 
improvement. Program staff with these skills will be able to monitor the program and make 
refinements and modifications leading to sustained programming. 

However, not all ECB work is intentionally requested by an organization that needs the 


support. In some cases, the work may be initiated by an external evaluator who sees the need to 


build ECB within the organization to support the evaluation objectives of a grant program, 
especially when sustainability planning is one of the grant objectives. The authors’ experiences 
with the TACCCT program evaluation provide one example of evaluators embedding ECB work 
within evaluation frameworks and processes to ensure that grantees (i.e., CTCs) have the 
capacity to continue the evaluation beyond the life of the grant. 

TAACCCT Overview 

The TAACCCT grant program, funded by the U.S. Department of Labor (DOL), was a 
$2 billion federal workforce investment over fiscal years 2011-2014. Approximately $500 
million was provided annually over the four grant rounds. The grant aimed to help institutions of 
higher education (IHEs) across the nation increase their capacity to provide education and 
training programs for in-demand jobs. The grant was designed to help IHEs and their partners 
develop new undergraduate education and career training program strategies; improve existing 
program strategies; or to replicate existing designs, program development methods, and/or 
delivery strategies that had demonstrated evidence of successful implementation. According to 
the DOL (2013, para. 3), the TAACCCT program’s goal was to: 

[h]elp ensure that our nation’s higher education institutions are able to help the Trade 

Adjustment Assistance (TAA)-eligible workers and other adults acquire the skills, 

degrees, and credentials needed for high-wage, high-skill employment while also meeting 

the needs of employers for skilled workers. 

In total, the DOL funded 256 three- to four-year grants; these were provided to IHEs 
offering programs that could be completed in two years or less (Mikelson, Eyster, Durham, & 
Cohen, 2017). Of the grant awards, 82% were two-year, public, degree-granting institutions 
(Cohen, Mikelson, Durham, & Eyster, 2017). 

As specified in the TAACCCT conceptual framework (Mikelson et al., 2017), the 


grantees were expected to build their organizational capacity through the implementation of 


TAACCCT activities to support long-term outcomes (e.g., effective training programs at 
community colleges with the capacity to serve students), and ECB was one of the focus areas of 
organizational capacity building (DOL, 2011, 2012, 2013, 2014). Specifically, grantees were 
expected to develop a plan to address “how data will be used to determine which strategies and 
activities were effective, and explain how they will integrate effective practices into core 
programs to enact broader institutional improvements” (DOL, 2012, p. 46). This expectation 
assumes that the grantees have or will build the capacity to collect, analyze, and use data to 
inform decision-making throughout the life cycle of the grant. Additionally, the grantees needed 
to have the capacity to continue the evaluation after the grant concluded to ensure that effective 
practices continued to be implemented as intended, broader institutional improvement was 
possible, and the program continued to have a positive impact. 

To support this expectation, beginning with the Round 2 grantees, TAACCCT programs 
were expected to work with a third-party evaluator to design and execute a rigorous evaluation 
plan that addressed both formative (i.e., implementation) and summative (i.e., an 
impact/outcome study) evaluation of the program. By Round 3, grant applicants were 
encouraged to collaborate with a third-party evaluator to develop the evaluation plan during the 
application phase, and were required to submit a 20-page detailed evaluation plan with very 
specific elements to be addressed once funded (Mikelson et al., 2017). The evolution of the 
TAACCCT grant application requirements highlighted the importance and value of evaluation in 
federally funded grant programs. As for the third-party evaluators, the authors also saw the need 
to incorporate and embed ECB strategies in the evaluation planning and scope of work to support 
the evaluation during the performance period (1.e., short-term goal) and sustain evaluation 


practice and culture after the grant ends (i.e., long-term goal). 


ECB Strategies Within TAACCCT Evaluation Framework 

ECB can be delivered in multiple ways. Usually it is a combination of strategies selected 
based on the areas of need that are driven by the motivations, assumptions, and expectations of 
ECB (Preskill & Boyle, 2008). Within the TAACCCT evaluation framework, the assumption 
was that evaluation provides data to inform continuous improvement and to understand program 
effectiveness. Hence, evaluation adds value to the grant-funded program. The motivation of ECB 
was driven by the need to meet accountability requirements, including the collection, analysis, 
and reporting of a set of complicated, pre-defined performance indicators. The expectations were 
that ECB would increase participating staff members’ knowledge, skills, and attitudes toward 
evaluation (i.e., short-term outcomes of ECB) and participating staff would continue to evaluate 
the progress and outcomes following the conclusion of grant funding (1.e., long-term outcomes of 
ECB). Based on these premises, as shown in Table 1, the authors identified a list of requisite 
knowledge, skills, and attitude areas that the TAACCCT grantees needed to have or should have 
been developing through ECB during the grant to help them understand and support the 
evaluation during the grant period and to enable them to continue evaluation work beyond the 


life of the grant. 


Table 1. Areas of Needs Checklist 


Areas of Needs 


Knowledge. Does the grantee understand... 


Evaluation design, objectives, and goals? 


(| Performance measures as defined by the DOL? 


Skills. Is the grantee able to... 


Areas of Needs 


(| Collect the data for the performance measures? 


(| Analyze the data to report on performance measures? 


|| Establish a data-sharing agreement with workforce agencies to secure wage and 


employment data? 


‘| Use data to inform decision-making? 
Attitudes—Does the grantee... 


Believe that evaluation provides useful information? 


_| Act upon the data-driven recommendations? 


Completing the checklist can assist in finalizing an ECB plan, but before finalizing the 
ECB plan, an ECB baseline assessment should be conducted to gather information to determine 
the priority areas and appropriate ECB strategies. ECB assessment can be conducted through 
formal data collection (e.g., surveys, focus groups, interviews) or informal assessment during the 
planning stage (e.g., an evaluation kickoff meeting). The authors used the latter approach to 
engage key project staff in the process of identifying the gaps and selecting the most appropriate 
ECB strategies to fill them. Based on the initial assessment, the authors planned and 
implemented a combination of six ECB strategies across eight TAACCCT projects. As shown in 
Table 2, some strategies were universal to all projects—such as written materials, meetings, and 
involvement in the evaluation process—while other strategies were targeted and individualized, 
depending on the needs and the size of the project. Examples of each type of strategy are 


discussed below. 


Table 2. ECB Strategies Embedded Within TAACCCT Evaluation Process Across Eight 


Projects 
Projects 
ECB Strategy 

D £E 
Provision of written materials 5 ai (ie ae > Vig (ne a LE 
Facilitation of meetings Ke | me a | ee | Re NI 
Involvement in an evaluation process > eam (ie. am > RV. Coll (>, nd (ik Sad Ue all. 
Provision of technical assistance Ks |. eS |i ROSH | OR Ko) xX 
Training Ke | RS | ees, | ee x 
Technology xX 


Written materials. Realizing that project staff may come from diverse backgrounds and 
have various experiences with and knowledge of evaluation, written materials were provided to 
grant project staff to address gaps they may have in knowledge regarding evaluation design, 
objectives, and goals as well as performance measures that were specified by the DOL. These 
written materials were distributed to appropriate project staff to ensure understanding of the 
various elements of the evaluation as well as the processes and procedures that needed to be 
implemented to collect reliable and valid data to inform performance measures. These 
documents also were distributed to new hire(s) during the onboarding process in CTCs. Other 
examples of written materials included rapid turnaround reports which provided timely data to 
inform continuous program improvement; an annual progress report that triangulated all data 
collected from various sources provided data-driven recommendations to inform program 
adjustments; and a final evaluation report focusing on program impact and outcomes. The final 


reports were intended to increase project staff's understanding of how the program works to 


support project outcomes, and to cultivate their evaluative thinking by using data to inform 
programming (Patton, 1997). 

Meetings. Regular meetings were one of the main ECB strategies that the authors used to 
build project staff's evaluation capacity in the areas of knowledge and attitude. For instance, a 
kickoff evaluation meeting was held at each CTC’s campus during the start-up phase. The 
purposes of the meeting were to establish mutual understanding regarding evaluation designs, 
objectives, and goals, and to identify areas of needs for evaluation capacity building. Following 
the kickoff meeting, regular monthly conference calls were held to discuss progress on and 
challenges to evaluation data collection and to review findings from the rapid turnaround reports. 
At these meetings, project staff gained a better understanding of the evaluation processes and 
learned to use data to inform their programming by reviewing the data presented and interpreting 
the data to make it meaningful for their program. Following the delivery of the annual evaluation 
report, a debriefing meeting was held either onsite or via webinar to review evaluation findings 
to date and facilitate discussions regarding the recommendations. Evaluators also helped the 
project team develop an action plan to support program improvement. These processes helped to 
strengthen project staff's knowledge and attitude regarding data use and put the data-driven 
recommendations into action. 

Involvement in an evaluation process. The most effective approach to increase project 
staffs skills in evaluation was to directly involve them in the evaluation process. Project staff 
were directly involved in the evaluation process in several ways across all TAACCCT grant 
programs. For instance, project staff were given opportunities to review and provide feedback on 
all data collection instruments before they were finalized. Additionally, project staff were 


involved in data collection activities by assisting with survey administration (distributing online 
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and/or paper surveys to survey participants) and facilitating onsite data collection. These 
processes gave project staff a sense of empowerment regarding their contribution to the 
evaluation and increased project staff's firsthand experiences with evaluation processes. As 
described previously, project staff also were involved in interpreting findings and participating in 
meetings to frame recommendations and create actionable strategic plans for program 
improvement. 

Training. Trainings were differentiated to address TAACCCT grantees’ unique needs in 
advancing their evaluation-related knowledge and skills. Topics covered in these trainings 
included understanding and interpreting TAACCCT performance measures, understanding and 
conducting participant tracking to report on TAACCCT performance measures, and 
understanding impact study designs as well as the collection of data to inform impact analyses. 
In conjunction with the trainings, written materials, such as those referenced above, were 
distributed to attendees. All trainings delivered via webinar were recorded so that project staff 
could review the webinars as needed and new hire(s) could view them as a part of the 
onboarding process. 

Technical assistance. One of the primary TAACCCT grant requirements was to collect 
participants’ employment and wage data. Per the DOL guidelines, the preferred data source was 
the workforce data collected by the local/state workforce agency. To access the data, CTCs were 
required to establish data sharing agreements with their local/state workforce agency. However, 
the availability of these data depends on state regulations regarding their release, with some 
states only sharing workforce data with federal agencies and others only providing aggregated 
data. With these restrictions, the processes and procedures to collect employment and wage data 


varied greatly. This issue became much more complicated when the grants involved multiple 
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institutions across different states (for instance, a consortium of higher education institutions). 
The evaluator’s role in this process was to assist with the drafting and understanding of a data 
sharing agreement that complied with state regulations and to provide an alternative plan to 
collect employment and wage data when workforce data were unavailable. Although not 
common, some CTCs who had experience with federally funded grant programs opted to 
undertake most of the evaluation themselves and hired an external evaluator to provide technical 
assistance. In this situation (one of the authors’ projects), the assistance provided to the CTC 
included the review and validation of the CTC’s evaluation processes and procedures as well as 
data interpretation and data use, to ensure CTC’s success in carrying out the evaluation. 

Technology. One of the TAACCCT projects was a large consortium that involved five 
institutions across five states, which meant the evaluation involved five different institutional 
data systems and five different states’ laws regarding workforce data. To ensure consistency in 
data collection and data management, the evaluation team developed a web-based technology 
system referred to as the Evaluation Data Management System (EDMS) that allowed each CTC 
to import and enter its own institutional data and workforce data while adhering to the guidelines 
that were established at the consortium level. Each CTC was also able to generate its own 
institutional reports to support program monitoring and improvement purposes. The processes of 
learning and using EDMS facilitated project staff's skills in data collection and use of data to 
inform programming. While technology made data more easily accessible to each CTC and 
consortium lead, it also required ongoing support and training to ensure data quality and 
consistency. 


Challenges 
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While ECB strategies implemented were guided by the areas of need based on the initial 
assessment with each institution, implementation of these strategies was sometimes challenging 
within the context of CTCs and external grant funding. One of the primary challenges is the lack 
of experience among most CTC staff in implementing a comprehensive evaluation. For some 
grant project staff, evaluation was a new concept. Although receptive to the idea, they faced a 
learning curve in understanding the nature and conduct of evaluation. Other project staff did not 
see the value; focused on the accountability aspects of evaluation to simply meet reporting 
requirements without understanding that continuous program improvement includes developing 
evaluation systems and processes that can enable the program to improve and be sustainable. 

Another challenge we encountered as third-party evaluators was access to the 
institutional research staff. In the authors’ experience, the CTC research staff may not be an 
integral part of the external grant project. Oftentimes, the research staff provide data requested 
by the project staff and, in turn, project staff shared the data with the evaluator. The challenge is 
that the external evaluator may not be in direct communication with the CTC research staff and 
therefore the data may not be provided in a way that can be analyzed to address the grant 
evaluation. Because of their multiple other responsibilities and responsibility to provide federal 
and state data to CTC leadership, the research office may not have the time or resources to 
devote to an external project. 

Additionally, the TAACCCT grants funded many positions that CTCs did not have prior 
to the grant (e.g., advisors and recruiters). Although not atypical with grant-funded programs, 
staff turnover was an issue. For project staff, they found they had to train new people on a 
regular basis, which caused inconsistency and delays in program implementation. For the 


evaluators, this also meant having to inform new staff about evaluation processes and 
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requirements, creating potential for inconsistency and delays with data collection. These issues 
were mitigated by creating durable ECB materials such as recorded webinars and written pieces. 

Lastly, the authors found that the sustainability of the evaluation processes and 
procedures established during the grant often dissipated once the grant ended. We believe this 
was primarily because most of the project staff who were trained in data collection and data use 
transitioned to different positions during the project close-out period. These personnel, including 
project managers, were not retained in their positions due to lack of funding after the grant. To 
overcome this issue, CTC leaders ought to engage staff early and continuously throughout the 
grant period so that they are positioned to be thinking—throughout the grant—how they can 
devote the necessary resources to support the important work (staff and activities) established 
with grant monies. 
ECB Planning Checklist 

Based on the authors’ experiences with the evaluation of TAACCCT projects, we 
constructed an ECB planning checklist for ECB practitioners (see Table 3). This reflects 


important considerations to build CTCs’ evaluation capacity throughout the grant process. 
Table 3. ECB Planning Checklist 


Checklist 


Start-up Phase 


‘| Convene CTC stakeholders for a project evaluation kick-off meeting. We found it 


essential to facilitate a kick-off meeting with CTC stakeholders, including project staff, 


institutional leadership and institutional research staff, at the beginning of the evaluation. 


Ultimately, this foundational meeting should encourage ownership and buy-in of the 
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Checklist 
evaluation process so that all stakeholders see the value of it. The third-party evaluator can 
also assess the areas of need related to ECB during the meeting by directly engaging the 


stakeholders in the process. 


Engage CTC institutional research office staff in the evaluation. Including institutional 


research staff in the evaluation taps into existing CTC expertise. This is especially true 
when it comes to identifying data that are already being collected that may be pertinent to 
answering the evaluation questions and developing additional systems or processes to 
manage the data. These connections should be built during the early planning stage, and 
evaluators should collaborate with the research staff, when appropriate, to create a tracking 
system that meets the needs of the grant. Evaluators can also use this as an opportunity to 
encourage leadership and research staff to improve the quality of their institutional data 


and databases. 


Implementation Phase 

(| Usea collaborative, participatory approach to involve project staff in the evaluation 
process. Collaborating with the staff to embed evaluation into the culture and ongoing 
operations of CTCs is essential to post-grant evaluation sustainability. The evaluator 
should build time into the evaluation plan to work with staff on creating evaluation 
frameworks, processes, and a strategic plan; implementing evaluation policies and 
procedures; finding resources dedicated to evaluation; understanding how to use evaluation 


findings; and integrating a knowledge management evaluation system. These activities can 


LS 


Checklist 
be completed throughout the grant life cycle through meetings (e.g., face-to-face or via 
webinar). Each meeting should have a well-defined agenda with established outcomes. 
Continue to garner institutional leadership support throughout the project period. 
CTC leadership should be engaged continuously throughout the evaluation to deepen their 
understanding of importance of evaluation and planning for long-term sustainability. For 
instance, hosting an annual debrief to present evaluation findings provides an opportunity 
for CTC leaders to engage in discussions and planning for continuous program 
improvement. Through guided discussions, project staff may gain stronger institutional 
support and access to more resources to build long-term sustainability. Moreover, engaging 
leadership may help to embed a positive culture of ECB that staff are excited to maintain 
on their own. 
Document and maintain institutional history record. Document everything that is being 
done so there is an historical record for the institution. This should be started at the 
beginning of the project so everyone involved in the project has the same understanding of 


processes and a reference guide. 


Close-out Phase 


(| Convene a transition meeting with CLC leaders, project staff, and institutional 


research staff. The purpose of the meeting is to assist the institution in identifying 


elements of the evaluation plan that can be implemented after the grant ends so that the 


implementation of the program can continue to be monitored and its impact assessed. A 
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Checklist 


feasible plan to continue collecting, analyzing, and reporting data to relevant stakeholders 


should be crafted at this stage. 


Conclusion 

What we describe is derived from our experiences with one federally funded grant 
program (TAACCCT); however, we believe that ECB can and should be applied to any grant, 
program, or project has been implemented by a CTC. The principles espoused in this paper 
transcend all programs implemented with defined goals and objectives. When program staff “do” 
evaluation simply because it is a grant requirement, it loses its potential value. To make 
evaluation meaningful, data need to be interpreted and used to monitor progress on goals and to 
make improvements. Evaluation reports “sitting on bookshelves, collecting dust” are 
meaningless. ECB provides program staff with an opportunity to understand and learn the value 
of systematic, informed data collection, interpretation, and application—evaluation—for 


program improvement and the documentation of successes for stakeholders and funders. 
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